The Layershift Logbook

Not every AI bot is the same problem. Some make your content discoverable in search and AI answers. Some act on a real person's behalf, fetching a page because somebody asked an AI assistant to do something with it. Others exist purely to train a model, with no visit at the end and no benefit back to you. Treating all three as one "AI bot" toggle, block everything or allow everything, throws away the information you need to make a sensible decision.

Robots.txt tells cooperative crawlers what you want. It doesn't enforce anything: a robots.txt file is a request, not a lock, and it only works on crawlers that read it and identify themselves honestly. Our earlier post in this series covers whether to allow Search, Agent and Training crawlers; the bot-by-bot guide covers what each named crawler does.

BotStopper and Cloudflare Bot Management get compared because they're both trying to solve the same underlying problem: verifying which bots are worth letting through and stopping everything that robots.txt alone can't. But they're not built to solve it at the same scale.